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Department  of  Defense’s  Waiver  of  Competitive  Prototyping  Requirement  for  the  Air 
Force’s  F-15  Eagie  Passive/ Active  Warning  and  Survivabiiity  System  (EPAWSS)  Program 

The  Weapon  Systems  Acquisition  Reform  Act  of  2009,  as  amended  (WSARA),  requires  the 
Secretary  of  Defense  to  modify  guidance  to  ensure  that  the  acquisition  strategy  for  each  major 
defense  acquisition  program  provides  for  competitive  prototypes  before  Milestone  B  approval — 
which  authorizes  entry  into  system  development — unless  the  Milestone  Decision  Authority 
waives  the  requirement.''  Competitive  prototyping,  which  involves  commercial,  government,  or 
academic  sources  producing  early  prototypes  of  weapon  systems  or  critical  subsystems,  can 
help  Department  of  Defense  (DOD)  programs  reduce  technical  risk,  refine  requirements, 
validate  designs  and  cost  estimates,  and  evaluate  manufacturing  processes  prior  to  making 
major  commitments  of  resources.  It  can  also  help  reduce  the  time  it  takes  to  field  a  system,  and 
as  a  result,  reduce  its  acquisition  cost.  WSARA  states  that  the  Milestone  Decision  Authority  may 
waive  the  competitive  prototyping  requirement  only  on  the  basis  that  (1)  the  cost  of  producing 
competitive  prototypes  exceeds  the  expected  life-cycle  benefits  (in  constant  dollars)  of 
producing  such  prototypes,  including  the  benefits  of  improved  performance  and  increased 
technological  and  design  maturity  that  may  be  achieved  through  competitive  prototyping;  or  (2) 
but  for  such  a  waiver,  DOD  would  be  unable  to  meet  critical  national  security  objectives. 

WSARA  also  provides  that  when  a  Milestone  Decision  Authority  authorizes  a  waiver  of  the 
competitive  prototyping  requirement  on  the  basis  of  what  WSARA  describes  as  “excessive 
cost,”  the  Milestone  Decision  Authority  is  required  to  submit  notification  of  the  waiver,  together 
with  the  rationale,  to  the  Comptroller  General  of  the  United  States  at  the  same  time  it  is 
submitted  to  the  congressional  defense  committees.  WSARA  further  provides  that  no  later  than 
60  days  after  receipt  of  a  notification  of  a  waiver,  we  are  to  review  the  rationale  for  the  waiver 
and  submit  a  written  assessment  of  that  rationale  to  the  congressional  defense  committees.^ 


Vub.  L.  No.  111  -23,  §  203(a),  as  amended  by  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year 
2011,  Pub.  L.  No.  111-383,  §  813.  DOD  modified  its  guidance  related  to  the  operation  of  its  acquisition  system 
through  Directive-Type  Memorandum  (DTM)  09-027,  “Implementation  of  Weapon  Systems  Acquisition  Reform  Act  of 
2009,”  (Dec.  4,  2009,  incorporating  Change  4,  Jan.  11, 2013).  DOD  issued  an  Interim  Instruction  5000.02,  “Operation 
of  the  Defense  Acquisition  System,”  on  November  25,  2013  which  incorporated  and  cancelled  DTM  09-027.  DOD 
reissued  the  interim  Instruction  to  update  DOD  policy  on  January  7,  2015.  A  major  defense  acquisition  program  is  a 
program  identified  by  DOD  with  a  dollar  value  for  all  increments  estimated  to  require  eventual  total  expenditure  for 
research,  development,  test,  and  evaluation  of  more  than  $480  million,  or  for  procurement  of  more  than  $2.79  billion, 
in  fiscal  year  2014  constant  dollars.  The  Milestone  Decision  Authority  for  major  defense  acquisition  programs  is  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics;  the  head  of  a  DOD  component;  or,  if 
delegated,  the  component  acquisition  executive. 

2pub.  L.  No.  111-23,  §  203(b)(1)  &  (2). 
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On  June  24,  2015,  we  received  notice  from  DOD  that  it  had  waived  the  competitive  prototyping 
requirement  for  the  Air  Force’s  F-15  Eagle  Passive/Active  Warning  and  Survivability  System 
(EPAWSS)  program.  The  F-15  Eagle  is  a  fighter  aircraft  designed  to  permit  the  Air  Force  to  gain 
and  maintain  air  supremacy  over  the  battlefield.  Its  current  tactical  electronic  warfare  system 
was  fielded  with  the  aircraft  starting  in  the  1970s.  EPAWSS  is  intended  to  replace  and  upgrade 
the  electronic  warfare  system  on  fielded  F-15C/E  aircraft.  It  is  expected  to  improve  the  aircraft's 
internal  self-protection  electronic  warfare  systems,  thereby  enhancing  the  F-15’s  ability  to 
detect,  identify,  locate,  deny,  degrade,  disrupt,  and  defeat  air  and  ground  threats  to  the  aircraft. 
EPAWSS  consists  of  four  major  subsystems.  These  include  the  radar  warning  receiver,  the 
electronic  countermeasure  processer,  the  countermeasure  dispenser  system,  and  the  fiber-optic 
towed  decoy.  These  capabilities  will  be  fielded  in  two  phases  or  increments.  The  program 
received  approval  to  enter  technology  development  in  August  2015. 

In  this  report,  we  assess  DOD’s  rationale  for  waiving  the  competitive  prototyping  requirement  for 
the  EPAWSS  program  and  the  analysis  used  to  support  it.  To  conduct  our  assessment,  we 
compared  the  rationale  in  the  waiver  to  the  WSARA  requirement  to  determine  the  extent  to 
which  the  waiver  is  consistent  with  the  statute.  In  addition,  we  reviewed  the  Air  Force’s  cost- 
benefit  analysis,  which  provides  the  data  and  assumptions  on  which  the  waiver  is  based,  the 
EPAWSS  acquisition  strategy,  and  other  relevant  documentation.  We  did  not  independently 
verify  the  Air  Force’s  data  on  cost  and  benefits.  We  also  submitted  written  questions  to 
EPAWSS  program  officials  to  clarify  information  in  program  documentation,  as  necessary. 

We  conducted  this  performance  audit  from  July  2015  to  September  2015  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and 
perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 


Results  in  Brief 

DOD’s  rationale  for  waiving  WSARA’s  competitive  prototyping  requirement  for  the  EPAWSS 
program  addresses  one  of  the  two  bases  provided  in  the  statute;  namely  that  the  cost  of 
producing  competitive  prototypes  exceeds  the  expected  life-cycle  benefits  (in  constant  dollars) 
of  producing  the  prototypes.  DOD’s  rationale  is  based  largely  on  the  acquisition  strategy  for  the 
program.  According  to  the  waiver,  the  program  is  to  leverage,  to  the  maximum  extent  possible, 
existing,  technically  mature,  non-developmental  components,  which  minimizes  the  potential  risk 
reduction  benefits  associated  with  early  prototyping.  In  the  waiver,  DOD  concluded  that  the  Air 
Force’s  cost-benefit  analysis  was  sufficient.  We  also  found  that  the  Air  Force’s  cost-benefit 
analysis  was  consistent  with  key  principles  in  DOD’s  policy  on  economic  analysis.^  The  Air 
Force’s  cost-benefit  analysis  supporting  the  waiver  examined  three  prototyping  scenarios 
including  the  system-  and  subsystem-level  prototyping  options  outlined  in  WSARA  and  DOD’s 
implementing  guidance.  The  Air  Force  concluded  these  strategies  would  increase  acquisition 
costs  between  $36.3  million  and  $116.3  million  (in  base  year  2014  dollars)  and  add  between  21 
and  41  months  to  the  program’s  schedule.  The  Air  Force  also  estimated  that  the  various 
prototyping  strategies  could  achieve  an  estimated  $6.5  million  to  $7.2  million  (in  base  year  2014 


^DOD  Instruction  7041 .3,  Economic  Analysis  for  Decisionmaking.  (Nov.  7,  1 995).  (Hereinafter  cited  as  DODI  7041 .3 
(Nov.  7,  1995)).  These  general  principles  state  that  each  feasible  alternative  for  meeting  an  objective  must  be 
considered  and  the  costs  and  benefits  associated  with  each  alternative  under  consideration  should  be  quantified. 
DODI  7041.3,  End.  3,  paras.  E3.1.1  and  E3.2.4  (Nov.  7,  1995). 
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dollars)  in  life-cycle  benefits  from  reduced  system  integration  risk  and  improved  performance. 
The  independent  Air  Force  Cost  Analysis  Agency  assessed  the  program  office’s  cost  estimate 
that  provided  the  cost  and  benefit  data  for  the  Air  Force’s  analysis,  which  helps  validate  its 
credibility. 


Waiver  Rationale  Is  Consistent  with  WSARA  and  the  Supporting  Analysis  Considered  a 
Reasonable  Set  of  Prototyping  Alternatives 

DOD’s  rationale  for  waiving  WSARA’s  competitive  prototyping  requirement  for  the  EPAWSS 
program  addressed  one  of  the  two  bases  provided  for  a  waiver  in  the  statute,  namely  that  a 
waiver  may  be  granted  if  the  estimated  costs  of  producing  competitive  prototypes  exceed  the 
expected  life-cycle  benefits,  including  the  benefits  of  improved  performance  and  increased 
technological  and  design  maturity  that  may  be  achieved  through  competitive  prototyping."^  DOD 
waived  the  requirement  to  competitively  prototype  the  EPAWSS  system  and  its  four  subsystems 
because  the  Air  Force  determined  that  the  prototyping  costs  greatly  outweighed  the  benefits. 

DOD  and  the  Air  Force  reached  these  conclusions,  in  part,  based  on  market  research,  early 
technology  readiness  assessments,  and  contractors’  demonstrations  of  key  technologies  that 
showed  EPAWSS  performance  requirements  could  be  met  with  existing,  non-developmental 
technologies.  Relying  on  existing  technologies  is  a  key  tenet  of  the  program’s  acquisition 
strategy  and  minimizes  the  potential  risk  reduction  benefits  associated  with  early  prototyping. 
The  Air  Force’s  market  research  found  that  non-developmental  technology  already  in  use  in 
U.S.  and  foreign  military  systems  could  be  integrated  to  satisfy  EPAWSS’  requirements.  The 
program  also  performed  an  early  technology  readiness  assessment  to  identify  and  confirm  the 
maturity  of  the  technologies  required  to  meet  EPAWSS  requirements.  The  Air  Force  found  that 
two  of  the  EPAWSS  subsystems — the  countermeasure  dispenser  system  and  the  fiber-optic 
towed  decoy — would  not  benefit  from  prototyping  because  they  require  no  further  development 
due  to  the  availability  of  existing  technologies  that  have  been  flight  proven  and  used  in 
successful  mission  operations.  For  the  other  two  EPAWSS  subsystems,  the  program  conducted 
technology  demonstrations  that  provided  some  of  the  same  benefits  as  competitive  subsystem 
prototyping.  The  program  office  allowed  contractors  to  demonstrate  their  candidate  subsystems 
for  the  radar  warning  receiver  and  electronic  countermeasure  processer  in  an  Air  Force 
research  lab  that  replicated  a  representative  operational  environment.  These  demonstrations, 
which  included  independent  subject  matter  expert  review,  assessed  the  maturity  of  the  four 
critical  technologies  associated  with  those  subsystems.  Based  on  these  demonstrations,  the  Air 
Force  found  that  the  technologies  were  mature  and  that  hardware  and  software  present  in 
industry  can  meet  EPAWSS  requirements.  Since  the  time  of  the  cost  benefit  analysis,  the 
maturity  level  of  the  critical  technologies  of  the  selected  EPAWSS  solution  has  been  reassessed 
at  a  lower  readiness  level  due  to  changes  needed  to  adapt  the  existing  technologies  for  use  on 
the  F-15C/E.  The  modified  technologies  will  be  retested  in  a  relevant  environment.  This  does 
not  change  the  Air  Force’s  conclusions  in  the  cost-benefit  analysis  and  the  program  still  expects 
the  technologies  to  be  mature  prior  to  beginning  engineering  and  manufacturing  development. 


"^In  the  waiver,  DOD  stated  that  the  cost  of  competitive  prototyping,  a  single  prototype,  and  critical  subsystem 
prototypes  of  the  EPAWSS  program  before  Milestone  B  exceeded  the  expected  life-cycle  benefits.  WSARA  provides 
that  whenever  a  Milestone  Decision  Authority  authorizes  a  competitive  prototyping  waiver,  the  program  is  still 
required  to  produce  a  prototype  prior  to  Milestone  B  approval  if  the  expected  life-cycle  benefits  (in  constant  dollars)  of 
producing  such  prototype  exceed  its  cost  and  its  production  is  consistent  with  achieving  critical  national  security 
objectives.  Pub.  L.  No.  111-23  §  203(a)(3)(A). 
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In  its  waiver,  DOD  found  the  Air  Force’s  cost-benefit  analysis  reasonable  and  we  found  that  its 
approach  was  consistent  with  general  principles  in  DOD’s  policy  on  economic  analysis, 
including  consideration  of  each  feasible  alternative  for  meeting  an  objective  and  evaluation  of  its 
life-cycle  costs  and  benefits.  In  its  cost-benefit  analysis,  the  Air  Force  developed  three 
prototyping  scenarios,  including  the  system-  and  subsystem-level  prototyping  options  outlined  in 
WSARA  and  DOD’s  implementing  guidance,  and  compared  the  life-cycle  cost  of  each  scenario 
to  the  potential  lifecycle  benefits.  The  Air  Force  found  that  the  additional  costs  ranged  from 
$36.3  million  to  $116.3  million  with  lifecycle  benefits  of  only  $6.5  million  to  $7.2  million  (all  costs 
in  base  year  2014  dollars).  The  scenarios  included:  (1)  competitive  prototyping,  in  which  two 
contractors  would  build  prototype  hardware  and  software,  (2)  non-competitive  hardware  and 
software  prototyping  involving  one  contractor,  and  (3)  prototyping  of  the  four  critical 
technologies  associated  with  the  radar  warning  receiver  and  electronic  countermeasure 
processer  by  one  contractor.  Results  of  the  Air  Force’s  cost-benefit  analysis  are  summarized  in 
table  1 . 


Table  1:  Summary  of  the  Air  Force’s  Prototyping  Cost-Benefit  Analysis  for  the  F-15  Eagle  Passive/Active 
Warning  and  Survivability  System  (EPAWSS)  Program 

Dollars  in  base  year  2014  millions 

Prototyping  Scenario 

Estimated  additional 
acquisition  cost 

Estimated  life-cycle 
benefits 

Estimated  delays  in 
initial  operational 
capability 

Competitive  prototypes  of  EPAWSS  - 
two  contracts 

$116.3 

$7.2 

41  months 

Prototypes  of  EPAWSS  -  one  contract 

$38.3 

$7.2 

26  months 

Prototypes  of  critical  EPAWSS 
subsystems  -  one  contract 

$36.3 

$6.5 

21  months 

Source:  Air  Force  cost-benefit  analysis  in  support  of  the  EPAWSS  competitive  prototype  waiver.  |  GAO-1 5-800R 


The  Air  Force  estimated  the  costs  and  time  required  for  each  prototyping  scenario  based  in  part 
on  historical  data  from  programs  it  considered  to  be  similar.  The  costs  included  the  cost  of 
developing  and  producing  prototypes,  as  well  as  systems  engineering,  government  program 
management,  and  testing  costs.  In  the  competitive  prototyping  scenario,  the  additional  41 
months  includes  the  time  needed  to  revise  the  acquisition  strategy  and  request  for  proposal  to 
incorporate  prototyping,  select  the  prototype  contractors,  develop  and  produce  prototype 
hardware  and  software,  and  test  the  systems.  The  costs  and  delays  associated  with  the  other 
two  prototyping  scenarios  were  based  on  similar  cost  and  time  factors.  The  estimated  life-cycle 
cost  benefits  were  largely  the  same  in  each  prototyping  scenario.  According  to  the  Air  Force 
cost-benefit  analysis,  each  prototyping  scenario  could  reduce  or  eliminate  the  software 
development  and  system  integration  risk  associated  with  adapting  potential  EPAWSS  systems 
to  a  challenging  threat  environment.  The  program  office’s  cost  estimate  included  $6.5  million  to 
address  that  risk,  which  provided  the  basis  for  most  or  all  of  the  potential  life-cycle  cost  benefits 
of  prototyping.  The  independent  Air  Force  Cost  Analysis  Agency  assessed  the  program  office’s 
cost  estimate  that  provided  the  cost  and  benefit  data  for  the  Air  Force’s  analysis,  which  helps 
validate  its  credibility. 
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Agency  Comments  and  Our  Evaluation 

We  provided  a  draft  of  this  report  to  DOD  for  comment.  In  response,  the  Air  Force  provided 
technical  comments,  which  we  incorporated. 


We  are  sending  copies  of  this  report  to  interested  congressional  committees,  the  Secretary  of 
Defense  and  the  Secretary  of  the  Air  Force.  In  addition,  the  report  will  be  available  at  no  charge 
on  the  GAO  website  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions,  please  contact  me  at  (202)  512-4841  or 
sullivanm@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs 
may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  were  Ron  Schwenn,  Assistant  Director;  Pedro  Almoguera;  Kristine  Massinger;  Laura 
Holliday;  and  Wendell  K.  Hudson. 


Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 


(121289) 
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This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  published  product  may  be  reproduced  and  distributed  in  its  entirety 
without  further  permission  from  GAO.  However,  because  this  work  may  contain 
copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be 
necessary  if  you  wish  to  reproduce  this  material  separately. 


GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 
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Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 
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